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12.0Inlroducrioit 

The pcrceptron and the convergence theorems or Chapter 1 1 are 
related to many other procedures thai arc studied in an extensive 
and diKordcrly literature under such titles as learning machines. 
MOimSOF LEARNING, INFORMATION RETRIEVAL. STATISTICAL DE- 
CISION theory, pattern recocnition and many more. In this 
chapter we will study a few of ihesc to indicate points of contact 
wilh (he pcrceptron and to reveal deep differences. We can give 
neither a fully rigorous account nor a unifying theory of these 
topics this would go as far beyond our knowledge as beyond the 
scope of this book. The chapter is written more in ihe spirit of 
inciting students to research than of offering solutions to prob* 
loan 

12.1 Informition Retrieval nnd tnductiir Inference 

The pcrceptron training procedures (Chapter 1 1) could be used to 

construct a device that operates within (he following pattern of 

behavior 





query ■ otdt 




Retrieve! 
Procedure 




Memory I 



Durinj) a "filing" phase, the machine is shown a "data set" of 
jr-dimcnsional vectors-one can ihink of ihetn as.n-bil binary 
numbers or as points in n-space. Later, in a "finding** phase, the 
muchinc must be able to decide which of a variety of "query" 
vectors belong to the data set* To generalise this pattern we will 
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uscihctcrmA^ Tor an algorithm that examines elements of a data 
set to modify che information in a memory. A** is designed to pre- 
pare the memory for use by another procedure* Af* -f which will 
use the information in the memory to make decisions about query 
vtttM 

This chapter will survey a variety of instance* of this general 
scheme We will begin by relating the perceptrox procedure to 
the simplest such scheme: in the cOMPLEiTJ storage procedure 
A h „ merely copies the data vectors, as they come, into the 
memory. For each query vector, A M searches exhaustively 
through memory to see if it is recorded there. 

I2.L1 Camparinj" miWTHN with COMPUTE STORAGE 
Our purpose is to illustrate, in this simple case, some of the ques- 
tions one might ask to compare retrieval schemes: 

Is the procedure univfrxal? The perccpiron scheme works per- 
fectly only under the restriction that the data set is linearly 
scp.ir.iblc. Compi.etf storage is universal: it works for any 
data set* 

Ho* much memory is required* CoimvT* storage needs a mem- 
ory large enough to hold the full data set. PtRCEpTRON. when ii 
1* applicable, sometimes has a summarizing eflccl. in that the 
information capacity needed to store its coefficients |oJ it sub* 
stantiallv le.ts than that needed to store the whole data set. We 
have seen ($10.21 that this is not generally true; the coefficients 
for i- rmn may need much more storage than does the list of 
accepted vectors. 

Nov quietly dors A tl *, operate? The retrieval scheme— exhaustive 
search specified for coMPitrr storaGC is very alow (usually 
slower than pi-rccptron's A,„a, which must also retrieve all its 
coefficients from memory). On ihc other hand, very similar pro- 
cedures could be much faster. For example, if A* did not just 
store the daia set in its order of entry, but sorted the memory 
into numeric*! order then Ami could use a binary search, reduc 
ing the query-answer time to 

lofij(Matasct)} 
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memory references. We shall nudy (in §126) A<* algorithms (hat 
sacrifice memory size to obtain dramatic further increases in 
speed (by ihc socalkd hash*voding technique). 

Can the proceJure operate with same degree {perha/u measured 
proimhthssicotl) ) of success eveti when K m has seen only a subset 
of thr data set rait it a "data atmpfe"? Pf RCFPTItON might, but 
ihecoMPiFTf storaof algorithm* as described, cannot make a 
reasonable goes* when presented with a query vector not in the 
data sample. This deficiency suggests an important modification 
of the complete storage procedure: (el A*^*, instead of merely 
checking whether the query vector is in the data sample, find that 
member of the data sample closest to it. This would lead, on an 
a priori assumption about the "continuity" of the data set* to a 
degree of generalization as good as the pcrceplron's. Unfortu- 
nately the speed-up procedures such as hash-coding cease to be 
available and we conjecture (in a sense to be made more prectoc 
in §1 2.7.6) that the low is irremediable. 

Other considerations *e shall mention concern the operation of 
A likt We note that the rrKCEPTKON and the CQHPLCT1 stokagb 
procedures share the following feature*: 

Th cy act inc remental ly , that is, change the stored memory slighily 
as a function of the currently presented member of the data set. 

They openi tc in "rcnMimc" without using large quantities of 
auxiliary scratch-pad memory. 

They ca n accept the data set in _any_ordcr and arc tolerant of 
repetitions lhal cause only delay but do not change the final 
jiatc. 

On the other hand they dilfer in at least one very fundamental 
way: 

The ryrc epl ronS A (fc i* a "search procedure" based on feed back 
from iu own r esults. The complete storage file algorithm ts pas- 
sive. The advantage for the pcrceptron is thai under some condi- 
tions it finds an economical summarizing representation. The cost 
i* thai it may need to sec each data point many times. 

IZ.t.Z Multiple Classification Procedures 

It in a slight generalization of these ideas to suppose the data 

*el divided into a number of classes Fi..,.,Ft< The algorithm 
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An B is presented as before with members of ihc data set but also 
» ith indications of the corresponding claw. It constructs a body 
of stored information which is handed over to A l1f whose task 
is to assign query points to their classes using this information. 

Fiamplc: Wo have seen ($M.3.I) how to extend the concept of 
ihc perception to a multiple classification. The training algorithm. 
At*, finds k vectors Ai,...,A t , and Ar** assigns the vector ♦ 
ioF,if 

♦ • A, > * • A, (all t p* /), INNER PRODUCT 

Example: The following situation will seem much more familiar 
to many readers. If wc think of each class F, as a "clump** 
or "cloud" or "cluster" of points in sfc-spaee, then we can imagine 
that with each F,U associated a special point B, that is, somehow, 
a "typical" or "average" point* For example, B, could be the 
cemrr of gravity, thai is, ihe mm of all the vocton in F, {or, 
say. of just those vectors that so fur have been observed to be in 
f>y Then a familiar procedure is: * is judged to be in (hat 
Fj for which ihc Euclidean distance 

I* - M- 

is the swatlcst. Thnt is. each * is identified with the nearest B- 
point. 

Now this nearness scheme and the inner-product scheme 
might look quite different, but they are essentially the same! 
For we have only to observe that the set of points closer to 
une given point B (han to another 8} is bounded by a hypcr- 
plane (Figure 12.1). and hence can be defined by a linear incquaf- 
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ity. Similarly. Hie point* closest to one of a number of 8/* form a 
(convex) polygon (Figure 12.2) and this is (rue in higher dimen* 
atom, also. 



ii 




FiflHMlU 

Formally, we see this b> obicrving that 

I* - B,! 1 -|#p - 2#*Bj+M- 

Now, iT we can nssumt lhai all (he *>*s have the same length r*, then 
the riucl)dejndKtAnce{^)wiflbe3'ira//efrwhcn 

*-%- J|D,|*« ♦ .»,-*, * 

t* largest. Bui ihi* ha* c*actt) the inner-product, *f the "ihreihold" 
ii removed by £1 2.1 1'* To M thai the inner-product concept Jo*et 
nothing by requiring the t«\ (o have ihe same length, we add an earn 
dimension and replace each * * [y ***jby 



*| < #•- W -£_, *>ij 



m> th.it all ♦** have length L - *r We have to add one dimemton to 
the IT*, too. but can ahvay* tel its coefficient to lero. 

IMA Variety of Classification Algorithms 

Wc select, from the infinite variety of schemes thai one might 
use 10 divide a space into different cltisscs, a few schemes that 
illustrate aspects of our main subject: computation and linear 
tcparation< We *v ill summarize each very briefly here; the re* 
mamderof the chapter compares and contrasts some aspects of 
their algorithmic structures, memory requirements, and commit- 
ments they make about the nature of the classes . 
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Each of our models uses the same basic form of decision algo- 
rithm for A** In each case there is assigned lo each class F, 
one or more vceiors A,; we will represent this assignment by say- 
ing that A, is associated with F >M . Given a vector *. the decision 
rule Is always to choose that f M for which A,* * is largest. As 
noted in 51 2. 1 .2 this is mathematically equivalent to a rule that 
minimises I* - A, I or s*une very similar formula. 

For each model we must also describe the algorithm A**, that 
construct* Ihe A,S, on ihe basis of prior experience, or a priori 
information about the classes In (he brief vignettes below, the 
fine details of the A,* procedure* are deferred lo other sections, 

1Z.2J The KKtltfMN Procedure 

There is one vector A for each class F, + An, can be the procedures 
described in $11.1 for the 2-ctass case and in §11.4.1 for the multi- 
class case. 



12.2 .2 The *\\ts Unear Statistical Procedure 

Again we have one A, for each f r Am ts quite different, however. 

For each class F, and each partial predicate ^.define 



W U - log 



I - p<>r 



whcre/\ ; ts the probability that *i m UffMH that 4 is in f,> Then 
define 

Aj - (*,.*v*v — >■ 

We will explain in $12.4.3 the conditions under which this use 
of "probability" mukes sense, nnd describe some "learning" algo- 
rithm* that could be used to estimate or approximate the w v *s. 

The r,\vi : 5 procedure has ihe advantage that, provided certain 
statistical conditions arc satisfied, it gives good results for classes 
that litctwt linear!) separable. In fact it gives the lowcsi possible 
error rate for procedures in which An depends only on condi- 
tional probabilities, given Ihiit the <'s are statistically Independent 
in the sense explained in 312.4.2. It Ji astounding that this is 
achieved by n linear formula. 
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1 2.2.3 i l ■ aisr plasfs Procedure 

In different situations cither pirceptron or bayes may b* 
superior. Gut often* when ihttF/l arc not linearly separable, there 
will exist a set of A, vector* which will give fewer errors than 
cither of these scheme*. So define the best planes procedure to 
DM ihal set of A/s for which choice of the largest A,-* gives the 
fewest errors. 

By definition, bfst pi anfs \s always at least as good as bayes 
or pprcfptrox* This does not contradict the oplimatity of dayes 
since the search for the best plane uses information other than 
the conditional probabilities. Unfortunately no practically effi- 
cient A*;* t* known for discovering its A/s* As noted in §l2J f 
hill-climbing will apparently not work well because of the local 
peak problem. 

12*2**1 The .- -■'. I '. Pn i ■ i'i,rv 

In the schemes described up to now. we assigned exactly one A* 

vector to each F*cto*s, If we shift toward the minimum-distance 
viewpoint, this suggests thai such procedures will work satisfac- 
torily only when the F-cla&ws arc "localized" into relatively iso- 
lated, single regions -one might think of clumps, clusters, or 
clouds* Given this intuitive picture, one naturally asks what to do 
if an F -class* while not a neat spherical cluster, is nevertheless 
scm (localized as a small number of clusters or, perhaps, a snake* 
like structure* We could still handle such situations, using the 
least-distance Am, by assigning an A*vcdor to each subvluster 
CiT each F t and using several A*s to outline the spine of the snake 
To realize this concept, we need an A** scheme thai has some 
ability to analyze disiributions into eluslers. We will describe one 
such scheme, called MHMTA. in §12.5, 

IZ^UTItcM'AR'Nr M-icnnOK Procedure 

Our simplest and most radical scheme assumes no limit on the 
number of A*VCcUh*. Ar, stores in memory every * lhat has ever 
been encountered, together with the name of its associated F- 
elass. Given a query veclor ♦,>, we find which ♦ in the memory is 
closest to 4*u. and choose the F-elass associated with thai ♦♦ 

This is a very generally powerful method; it is very efficient on 
many sorts of cluster configurations; it never makes a mistake on 
an already seen point; In the limit it approaches zero error 
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except under r.iiher peculiar circumstances (one of which is dis- 
cu»cd in the following section). 

nearest neighbor has an obvious disadvantage — the very large 
memory required and a subtle disadvantage: there U reason to 
fuspcel thai it entail* large, and fundamentally unavoidable, com- 
putation costs (discussed in §12.6)* 

12*5 I learistic Geomplr) of Linear Separation Method* 
The diagrams of this section attempt to suggest some of the be- 
havioral aspects or the methods described in &12.4. To compen- 
sate for our inability to depict multidimensional configurations, 
we use two-dimensional multivalued coordinates. The diagrams 
may appear plausible, but they are realty defective images that 
do not hint at the horrible things that can happen in a space of 
many dimensions. 

Using this metaphorical kind of picture, we can suggest two kinds 
of situations which tend to favor one or the other of haws or 
rnu'EPTRON (see Figure 12.3). 



rncftfRM 




BArfS 




The iKYfS line in Figure 12*3 lends to He perpendicular to the 
line between the "mean" points of F, and F». Hence in Figure 
13.31a). we find ihat bavis makes some errors. The sets are* in 
fact .linearly separable, hence ptRCCPFKON, eventually, makes no 
errors at all. In Figure 12J(b) we find that SAVE* makes a few 
errors, juntas in !2J(a). Wcdon*lknow much about pfrcfptron 
in i.on*cparablc situations: it is clear that in some situations it 



<■ ■ 



a 



|I96] (12,3) Learning Theory 



w ill noi do aswcll a* nAvrs. By definition nest pianB, of course, 
docs ji least as well as cither bayes or pekckptron. 

From the Man, the very suggestion thai any of these procedures 
will be any good ill all amounts to an a priori proposal that the 
F< classes can be lilted into simple clouds of some kind, perhaps 
with a little overlapping, as in Figure 12*4. Such an assumption 





Figure 12.4 

could be justified by some reason for believing that ihc difference 
between F. and F. tire due to some one major influence plus 
a variety of smaller, secondary effects. In general fgrcfpiron 
tends to be sensitive to Ihc outer boundaries of the clouds, and 
relatively insensitive to the density distributions inside, while 
\\\\ I s weights all *Ks equally. In eases that do not satisfy either 
the single-cloud or the slight -overlap condition (Figure 12,5)* we 
con expect &ay» % to do badly, and presumably perceptron also, 
nesr fLANli can be substantially belter because it is not subject 
to Ihe bad influence of symmetry But finding the iCST piane is 
I 




Figure 12.5 
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ltkcl> in involve bad computation problems because of multiple, 
locally optimal "hills. 14 I : igurc \ 2.6 shows some of the local peaks 
for MOFT plane in the case of a bad "pafftyHfct" situation. Here, 
even isodata will do badly UltlOM it is allowed to have one A- 
vector for nearly every clump. But in the case of a moderate 
number of clumps, with an A s in each, r$0l>ATA should do quite 
well (Sec §125.) Generally, wc would expect PERCEPTION to be 
slightly better than lAYfit because it exploits behavioral feedback, 
worse because of undue sensitivity to isolated errors. 
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figure 1 2,6 

One would expect the nearfst neighbor procedure to do well 
under a very wide range of conditions. Indeed, NEAREST 
NHtiiinoft in the limiting case of recording all 4>\ w ith their class 
noma will do at least as well as any other procedure. There are 
conditions* though, in which nearest neighbor does not do so 
well until the sample ai/c is nearly the whole space. Consider, 
for example, a space in which there are two regions: 



P(4«F.W 




P(*«F.) * l-P'l 
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In the tipper region a friction p of the points arc in F. , and these 
arc randomly distributed in space, and similarly for F_ in the 
loucr region. Then if 3 small fraction of the points are already 
recorded, the probability that a randomly selected point has the 
same F as it* nearest recorded neighbor is 

j* + 1* - I - 2pf t 

while the probability of correct identification by bay£S or by 
best plane is simply p. Assuming that p > \ (if not. just ex- 
change/* and q)\\c sec that 



Error 



*wn**t 



< Error 



NLAUVr KIIGMMH 



< 2 x Erwr_r- 
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*o that nfarfst KHfitinoR i> worse than best plane, but not 
arhiirariiy worse. This clTeel will remain until so many point* 
have been sampled that there is a substantial chance that the 
sampled point has been sampled before* that is* until a good 
fraction of the whole space is covered* 

On the other side, to the extent that the "mixing" of F. and 
F. is ks$ severe (see Figure 127). the nearest neighbor wilt 
converge to very good scores as soon as there is a substantial 
chance of finding 0/1* sampled point in most of the microclumps. 




Figure \2J 

A tcry bad ease is a paritylikc structure in which nearest 
Nt-iuLtnoK actually doc* worse lhan chance* Suppose that 4 t F, if 
and only if ** ■* I for an even number of f*s. Then, if there 
are n ****. each $ will have exactly n neighbors whose distance d 
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KUbBetQ < d < I. Suppose that nil bul * fraction q of All pos- 
>ibJe *'s have already been seen. Then nearest neighbor will 
err on a given ♦ if it ha* not been seen (probability - q) but 
one of lis immediate neighbors ha* been seen (probability - 
I - */*). So the probability of error is >q(\ - q*h which, for 
large 4, can be quite near certainty. 

This example is. of course, "pathological." us mathematicians like 
to say, and NEARESf xfigudor is probably good in many real 
situations* Ms performance depends, of course* on the precise 
"metric" used to compute distance, and much of classical statis- 
tical technique is concerned with optimizing coordinate axes and 
measurement scales for related applications. 

Finally, wc remark that because the memory and computation 
costs for this procedure arc so high, it Is subject to competition 
from more elaborate schemes outside the regime of linear separa- 
tion - and hence outside the scope of this book. 

I 1.4 Decisions Qascd on Probabilities of P red iealc- Values 
Some of the procedures discussed in previous sections might be 
called "statistical" in the weak sense that their success is not 
guaranteed except up to some probability. The procedures dis- 
cussed in this section are statistical also in the firmer sense that 
they do not store members of the data set directly, but instead 
store statistical parameters, or measurements, about the data set. 
We shall analyze in detail a system that computes- or estimates— 
the conditional probabilities p„ that, for each class F,the predicate 
a*, has Ihc value I . It stores these pj* together with the absoluie 
probabilities p t of + being in each of the F/s. 

Given an observed *, the decision to choose an F, is a typical 
statistical problem usually solved by a "maximum likelihood" or 
Dayes-rulc method. It is interesting that procedures of this kind 
resemble very closely the percept ron separation methods. In fact, 
when we can assume that ihc conditional probabilities p v are suit- 
ably independent (§12.4.2) it turns out ihji the best procedure is 
the linear threshold decision wc called saves in §12 2,2. We now 
show how this comes about. 

12,-1.1 Maximum I ikclihuud and Base* Law 

In Chapter 1 1 wc assumed that each ♦ is associated with a unique 

F,. Wc now consider the slightly more general case in which the 
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same ♦ could be produced by events in several different F-classcs. 
Then, given an observed ♦ we cannot in general be sure which 
F v is responsible, bm we can sit best know the associated probabil- 
ities. 

Suppose lhal a particular ** ha* occurred and we want to know 
which F is most likely. Now if F f is responsible, then the "joint 
event" Fj A 4« has occurred: this has (by definition) probability 
fl*lF, A 'p?y Now (by definition of conditional probability) wc can 
write 

(PfF, A «„) - <P <F,) <<?(♦„ I ty. (I) 

Thai it, the probability that bolh F, and *o will happen toother is 
ci|ual io the probability that F, will occur multiplied by the prob- 
ability that tfFjtmws IB *iil ♦«- 

We should ehoote thai F, which gives Formula I the largest value 
because (hat choice corresponds to the most likely of (hose events 
lhal could have occurred; 

F,A •« F,A* -- F 4 A* - 

These are serious practical obstacles to the direct use of formula I. 
If there are many different **s it becomes impractical to store all 
the decisions in memory, let alone to estimate them all on the 
basis of empirical observation* Nor has the system any ability 
to guess about *Vs it has not seen before. We can escape all these 
difficulties by making one critical assumption— in effect, assunv 
ing <hc situation closely fits a certain model-that the partial 
predicates of * ^ {<p **) are suitably independent. 

1 2*4*2 Independence 

UpioiuvA hc nave suppteiscd the A"» of earlier c ha piers because wc did 
not care where the values of the »**£ came from. We brinj them hack for 
a moment m that we can five a natural contest to (be independence 
hypoihctit. 

We can evade the prohlems mentioned above if wc can assume 
that the tots v?*(A'> are statistically independent over each F-class, 
Precisely* this mea*s lhal Tor any *(X) - \>i(-V), . >*. . *p m {X)) wo 
can assert that, for each/, 

flWF,) *<P<^F,)x".*(P(<>jF,>. (2) 
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W; emphasize lh;ii ihis U a most stringent condition. For exam- 
ple, it ift equivalent to ihc assertion that: 

Given thai o ♦ is in a certain F*closs t if one is told also the values 
of some of the v's. this gives absolutely no further information about 
the values of the remaining *p*s. 

Experimentally, one would expect to find independence when the 
variations in the values of p*j are due to "noise" or measurement 
uncertainties within the individual ^-mechanisms (Figure 12 8) T 
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Figure 12,8 

For, to the eMcni that these have separate causes, one would not 
expeel the values of one to help predict the values of another. But 
if the variations in the ^*s arc due to selection of different X's from 
the same tV/ew, one would not ordinarily assume independence* 
since the value or each v tells us something about which X in F 
has occurred, and hence should help at least partly to predict the 
values of other tf's (sec Figure 1 2.9). 
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Figure \2S 

An extreme cm in pic of nnnimlcpcmlencc is the following: there 
.lrelwodsiiMci, F| and F ; . and two *\ defined by 



*r,(.V> - a pure random variable with &{*AX) » I) ■ J, 
It* value is determined by tossing a coin, not by X. 
iV . UiiX)HXi F„ 
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Then 
But 



Wl*i 






Notice that nvithtr *?, nor f 3 takm alont jriiw any information 
+htitc\rr about Ft Each appears (o be a random coin loss* Bat 
from holh one mm determine pcrfcclly whkh Fhasoccured* 

for v*i m +2 implies F(. 
while v ■. *• +: implies Fj I 

with absolute certainty. 



rfmark: Wc atiume only independence within each class F^ So tl X « 
not given, then kurninf one ^-vnluc rM help predict another. For ex* 
ample, suppose that 



vi 
*! 



#, - Oir-Tt Tu 
#, - I Vjr < Fj. 



Thejetwof'jareio fact independent on each F. Bui i(uc tint not know In 
iidvanft that X t F| hul were told ihnt * t * 0. we could indeed then 
predict thai <; ■ also, without this violating our independence as- 
sumption, (If we had previously been told that X « Fj. then we could 
atnviJy predict the value of ^*; in that case learning the value of &i 
would have no effect on our prediction of ^j.) 

I2.-IJ The maximum likelihood decision, for Independent ip% Is a 
linear threshold predicate! 

Auumc thai (he g»/s arc suiLMicalh independent Tor each F 
Define 

to -W> 

ft-W^-llFjJ, 



Suppose that wc have just observed a ♦ = (^ ^ H ), and we 

want to know whi:h F, was most probably responsible for this 
Then- according to Formula* I and 2, wc will choose that j which 
maximizes 
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n* rl' it 

. Pa * *v 
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Dccauic sums arc more familiar to us than products, we will 
replace these by their logarithms. Since log x increases when x 
docs, we still will xekel the largest of 



£ ft ■ W^ + [lof n -»- ^toffj. (3) 

7*i \ / 



Because the ri^hl-hnnd expression is a constant that depends only 
upon /. and not upon the experimental ♦, all of Formula 3 can 
Ik written simply as 

- * M *< + *j* (3') 

Rumple I: In the case ihat there arc just two classes. F, and F a , 
we can decide IbM V « F whenever 

SwjiW + ** > 2w.r** + ** 
that is. when 

2('i\i - *aVi > (*: - *<). (4) 

which has the form of a linear threshold predicate 
i> - (£*■*,> t\ t 

Thus wc have the rem art able result that Ihc hypothesis of inde- 
pendence among ihc #*s leads directly to the familiar linear 
decision policy. 

Example 2 (probaMiiie* of error): Suppose that for all f, /ta ■ g<j. 
Then a, is the probability that *,(.¥) - tf(^) and 9,1 is the prob- 
ability that «r,CV) * J(*V), that is, that *?, makes an error In 
(individually) predicting the value of^ - IX * F||. 
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Then inequality 4 becomes 
* Pi 



(*') 



Now obxrrvc thai (he (2*?, - 1) term has ihc effect of adding or 
subtracting v„ according Lo whether ^, ^ I or 0* Thu*, we can 
think of the h*** a* weight* to be added (according lo (he ur's) to 
one tide or the other of a balance: 





The log if>:!pi) « ihe %% a priori weight" in favor of F ; at the 
marl, and each w u - lt>|<^i/«i) is *•« " w *^nt or the evidence" 
lit hi ^ - I gives in favor of F|* 

Iti* quite remarkable that the optimal separation algorithm pven that 
the ./-probabilities aie independent bat the form (inequality 4) of i 
linear threshold predicate. Bui one mutt be sure to understand that if 
|t*,v? > *l i* ihe "optimal" predicate obtained by the independent* 
probability method, yet doc* not perfectly realise a predicate rf. ihb 
doe* not preclude Ihe exigence of a precise separation \Za*<e > #1 
uhiehalttay* agree* with *. [This is the situation suggested by Figure 
I? ?(a).| Tor inequality 4 i* "optimal" only in relation to all A*k pro* 
ccdurcs that u*v na information aibvr short the ewtditionaf ptobabijitits 
\p t \ and \p%i\* while a pcrccptron computes coeiTieienl* by a nonsumtical 
search procedure that i* *en*Hivc to individual events. 

Thun.if# i* in facl in /,(*! the pereeplron will eventually perform at 
least iuwcH a* any linear**! a tfrocal machine* The latter family can have 
the advamafc in w>mc ca*es : 

1. If J 1 M+) ihc lUlbtteri scheme may produce a good approaimtiB 
teparalion while the pereeplron might fluctuate wildly* 

2. The lime |0 achieve a iraliil level may be Jong Tor the perception file 
algorithm whteh ft basically a utriil *carch procedure. The linear-4laiis- 
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deal machine is basically more parallel, became it finds each coefficient 
in dependent of the oihcr*, and needs only a fair sample or Ihc F*f. 
(While supcinciairy pcrceplron coefficients appear lo be changed indi* 
viduatly* each decision about a change depends upon a test involving all 
I he coefficient*) 



12,4*4 Ijiyer-M a chines 

Formula J 1 iuggests ihc design of a machine Tor making our 

decision: 




D i* a device ihiit simply decides which of its inputs i$ the largest. 
Each ^device cmit< a ilamlard -sized rwltc (if &(X) - 1) when X 
» presented. 'Hie pulses arc multiplied by the h*„ quantilics as 
indicated, und summed ni the «-bo*cs. Tbc 9 } terms may be re* 
£ j nied ;is corrections for the extent 10 which the p^% deviate from 
a central value of ). combined wiUi (he a priori bias concerning 
F, itself. 

It it often dominie to minimi/c the ani\ of errors rather than simply 
the chance of error If wc define t\t lobe the cost of euctsing F 4 when it 
it really Fjlhai ha* occurred, inert il is easy to sho* that Formulas 1 and 
2 now lead to finding lite A that minimize* 



£<-,.- «,-n(? 



/ "■ 



* here 1 



II ^ v+ It it intemtinj* that this more complicated procedure 



abn lends iHclfto the multilayer Miueturc; 



-\o 
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It ought lo be piKMblc 10 devise a training algorithm to optimize the 
uoohlt in thi* using. *Jy. the magnitude of A reinforcement aignal to 
communicate to the net the co*t of an error We have noi investigated 

12.4.5 Prob*hlUty-c*timation procedures 

The A f * algorithm for ihc MTO linear-statistical procedure has 
to compute, or estimate, either the probabilities /r„ and fy of 
formula 3 or other statistical quantities such as "weight of evi- 
dence" ratios /p/(1 - pi Normally these cannot be calculated 
directly (because they arc* by definition, limits) so one must find 
estimators. The simplest way to estimate a probability is to 
rind the ratio !I/N of the number // of "favorable" events io the 
number S of all events in a sequence. If <? M is the value of * on 
the nh trial then an estimate of <*■(«? - I) after n trials can be 
found by the procedure: 



%r^r: 



RhPFAT: 



Set a to 0* 
Set ft to 1. 



5ct <* lo 



(n - 1)a 



•H 



Set* 10 « + I- 

OlQ 10 REPEAT 



which can easily be seen to recompute the "score" ll/N after 
each event, 

Tlii* procedure has ihc disadvantage that ; t has to keep a record 
of n\ the number of trials. Since a increase beyond bound. thts r 
would require tinlimiied memory* To avoid this, we rewrite the 






/I 
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above program*! computation in the form 

.«.(,-i.y.- 1 . + i.„- 

This suggests a simpler heuristic substitute: define 



,■> - 0. 



.hi 



(1 - .)«*-» + t * * w . 



(S) 



where t is a "small" number between and I. It fa easy to show 
that as w increases the expected or mean value of «**, which we 
will * rite as {<»*■'), approaches/* (that is. { y^ ) as a limit. For 



<«' 



- H - - 0>. 



end 



(..'*)- (I - .)(! - {1 - <))p f .;» 
-<!-<!— i?)p. 

■nd one can verify ih-il. fai nil «. 

W (I -(I - i)')p 



(asn 



') 



Thus, process 5 gives an estimation of the probability thai * m U 
A more detailed analytic would show how the estimate depends 
on (he events of the recent past, with the effect of ancient events 
decaying exponentially with coefficients like (I - 0* - *" 1 ** 

Because process 5 "forgets." it certainly does not make "optimal** 
use of it* past experience; but under some circumstances it will be 
able to "adapt" to ch:in£in£ environmental statistics, which could 
be a good thing, A* a direct consequence of the decay, our esti- 
mator has a peculiar property: its variance *>'" does not ap* 
: v, i .- /cro. In foci, one can show that, for process 5, 



r ; * 



p(l " /■> 



i - * 



<>u 
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and this, while nol «ro. will be very small whenever * is. The 
situation it thus quite different from ihc H/tf estimate—whose 
variance \**p[ I - p)f*t and approaches zero as n grows. 

In f.'K-i. we can use the variance to compare the two procedures: 
If wc "equate" the variances 






wc obtain 



1 



sxTpgcsiing thai ihe reliability of the estimate ofp given by process 
5 is about ihe same as we would gel by simple averaging of the 
last 2/t samples; thus one can think of the number l/r as cor- 
responding to a "lime-constant" for forgetting. 

Another estimation procedure one might consider is; 



ruruuzn 

FfiHATE 


Set a to anything, 

If^ - 1, set ft toa *■ I. 
IJ> - 0,setft to(l - f)a. 

CiO tO REPEAT. 





Convergence to ihr Fucd- Point 



Auta 



> 
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or. equivalent ly, one could write 

h w - (1 - 0«^ n + + m* 1 "' 1 )*"- 

It can be shewn that thishai an expected value, in the limit, of 

It it interesting that a direct estimate of the livelihood ratio is 
obtained by such a simple process as if * - I add 1, otherwise 
muitiptyby{\ - *}■ The variance, in case anyone cares, is 

... > . ■ , 

(i _ ,)' i - (i - *r 

12,4.6 Thr Samuel compromise 

In his classical paper about "Some Studies in Machine Learning 
using the Game of Checker*," Arthur L. Samuel uses an in- 
genious combination of probability estimation methods. In his 
application it occasionally happens that o new trident* term *, is 
intnhftteed (and an old one is abandoned hecausc it haft not been 
of much value in the decision process). When this happens lhe« is 
a problem ofprcvcntmg violent fluclualwns, because after one or 
a few [rfab the term's probability estimate will have a large 
variance as compared with older terms that have better statistical 
records. Samuel uses the following algorithm to "stabilize" his 
system: he sou a m to ] and uses 

■"•"-('- »)■" * h '"•"• 

tvherc A' is set according to the "schedule": 



v 



16 if ft < 32. 

2* if 2* 5 fl < J"* 1 and 3! < n 5 256, 

256 1 if l '256 Si * [ 



Thus, in the beginning the estimate is made as though the prob- 
ability had already been estimated to be J on the basis of several. 



k- 
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thai is> the order or 16, trials* Then in the u mfcWle* i period, the 
algorithm approximates ihe uniform weighting procedure. Finally 
(when n - 256) ihe procedure changes 10 Ihe exponential decay 
mode, with fixed jV.soihai recent experience can outweigh earlier 
results* (The use of the powers of two represents a convenient 
computer-program technique for doing this.) 

In Samuel's system, the terms actually used have the form we 
Tound in inequality 4* of § 1 2,4*3 

U m - i 

so that the "estimator" ranges in the interval -1 S p U] S + I 
and can he ircalcd us a "correlation coefficient*" 

12*4*7 A simple "sjnaplic" remforccr theory 

Let us in iikc .1 simple "neuron;!! model/* The model is to calimalc 

Pa m Pi&A^jt* using only informaiion about occurrences of 

I?, - II and of f* i F,l* Our model will have the following 

"anatomy' 1 : 




{O 



The bag B, contains a very high and constant concentration of a 
substance £. When ^ t or F, occur- -or "fire" the walls of the 
corresponding bags B t and/or C t become "permeable 1 * to E for a 
moment. IT^, atone occurs, nothing rcully changes, because d t h 
surrounded by the impermeable C, If F / alone occurs* C } loses 
some E by diffusion to the outside: In fad* if a is the amount of 
£ In C } it may be assumed (by the usual laws of diffusion and 
concent ration) to lose some fraction < of a: 

t ■ * * .- F, occurs and 

a -(I -o« .f |;, 

If both tf , and K, occur then approximately ihe same loss will occur 
from £ \ Simultaneously* an essentially eon* t a tic amount b will be 
"injetried" b> dilTusion from B t to C r So 

i ff « f \F t occurs and 

1 ** » J. 



'',: 
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(We can assume that b is constant because the concentration off 
ts very high in B t compared to thai in C,. One can invent any 
number of similar variations. Hn either case we gel 

a 1 * (I - t)a + ?b 

so that in the limit the mean of <* approaches b * p (a* can be 
seen from the analysis in jjl2.4,5). This is proportional to. and 
henecancMtmarorofp,, - PfolFjJ. 

Thuii the simple anatomy, combined with the membrane be- 
coming permeable briefly following a nerve impulse, could give a 
quantity that is an estimator of the appropriate probability. 

Mow could this representation of probability be translated into a 
useful neuronal mechanism? One could imagine all sorts of 
schemes: ionic concentrations- or rather, their logarithms! — 
could become membrane potentials, or conductivities, or even 
probabilities of occurrences or other chemical events. The "an- 
atomy** and "physiology" of our model could easily be modified 
to obtain likelihood ratios. Indeed, it is so easy to imagine 
variants- -the idea is so insensitive to details — that we don't pro- 
pose it seriously, except as a family of simple yet intriguing 
models that a neural theorist should know about. 



12.5 An, algorithm* for ihe isoiuta procedure 

In this section wc describe a procedure proposed by G, Ball and 

D. Hall lo delineate "clusters" in an inhomogencous distribution 

oTvcciors^ The idea is best shown by a pictorial example: imagine 

a two-dtmcniionalset ofpoints|*| that fall into obvious clusters, 

like 




<3> 
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Bcuin by placing a Tew "cluster-point*" A'* 1 into the space at some 
arbitrary locutions, say, near the center. We then divide (he set 
of*"* into iubieU R„ assigning each + lo ihc ttrarest A\ i] point 




Next, we replace each A! 1 ' by a new cluster-point Aj'* which is the 
mean or ccnicr of-^rjvuy of the 't*\ in R„ and then define R [*' to 
be the sci of +*s nearest lo A:: 




■y- 
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Repealing we get a new set of A/sand RjV. 




and n«i 



* 





; 







; 




-Z^ 




^\ 


K ' 


'A, 


t) 


\^ 


/ 


V ' 


t^l 


* ■ 


S 


_i^~ D ' 




^ 


t-* - — * -*' 




s 


j 


S N S *^ : 




* 
■ 







From now on, there is lilllc or no change; the cluiter-poinls have 
"found iheetuilcrs," 

Bait and lljil give a number of heuristic refinements for creating and 
tkfrlroying additional dirtier point*; for example, to add one if ihe 
variant of an R-»tf is "loo larfltT and to remove one if two »re "too 



..^v^ 
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<lo«*' together Of course, one can usually *W* two-dimensional 
elusion by inspection, but isodata b alleged to give useful fesulu in 
'■-dimensional problems where "inspection* 1 is out of the question. 

To use tlii* procedure in our comctf, W need some way lo com- 
bine its automatic classification ability with ibe information about 
ihcF*clnsscs. An obvious first step would be to apply it separately 
lo each F-doffc and assign all the resulting A's to thai class. Wc 
do not know much about more sophisticated schemes that might 
lead to belter results in the \ ip4 stage. 

12.5.1 An isooata corner^cucc theorem 

There fa a Iheorem jboul isodata (told to us by T« Cove*) that suggests 
that II leads lo some sort or local minimum. Let us formalize the pro- 
cedure by defining 

A H (*| - iheA'fihaiis nearest lot. " 



(If there arc several equidistant nearest A/*, choose the one with ihe 
smallest index.) 

«? - lheveiof*VorwhiehA w « - A 1 ?. 

A^-meanW 

Finally define a "score": 



**? 



-Wm: *<« >#»>,.- »J« ». 



until thercunorunherchangc.ihilis- until A l J° - A 1 * * " for alii. 
FftOOK 



-£!>-*! 



Mil 



' ■?• 



ZEi*-* 1 ;" 1 !' 



,,. 



because the mean (A 1 '* 11 ) of an> set of vectors (R* ; l a just the point 
that minimize* the squared -distance sum to all the points (of fi'f I And 
ihis is* in turn, 



EI 



- aT"| j - *>"*" 



.; 
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becausecnchntfiniwillbelrjnsrcrrcdlOin Rff * lf for « hich the distance 
it minimal, thai is, for illy, 

|#-Ap' r | * !•- A? I. 

CerolUrt t 7A<- irouenc* 0/ decrees (ng paiitivr numbers. |j***J approaches 
a //w//. If there it onty a finite wt i*/* j. fAe A*j must stop changing in a 
finite number of sups. 

For in (he finite case tnerc are *>nly a finite number or partitions J? 
possible 

12.5.2 Incremental methods 

In nnulog> to (he "reinforcement" methods in §12.4.5 wc can ap- 
proximate isooata by the following program: 



start: 
ifpfat: 


Choose a set of starting pointsA^ 

Choose a *♦ 

Find A($): ihc A, nearest to * t 
Replace A(*) by (1 - <)A(*) + * - *. 
Go tO HtPUAT- 

1* 





lib clear that (his program will lead to qualita lively the same sort 
or behavior, ihe A*s will lend toward the means of their R-/egiont. 
Bui, just as in §12.4, ihe process will retain a permanent sampling- 
and-fbrgcuing variance, with similar advantages and disad- 
vantages. In fact, all the Aft algorithms we have seen can be so 
approximated: there always seems to be a range from very local, 
incremental methods, lo more accurate, somewhat less "adap- 
tive" global schemes. We rmimc this discussion in §12.8. 



12.6 Time vs. Memory for F*»cl Matching 

Suppose that we are given a body of information —we will call 
it the diUa s*s in the form of V binary words each b digit* in 
length (Figure 13.10); one can think of them as V points chosen 
at random frorr a space of 2* points. (Take a million £r 2* words 
of length 100, for a practical example.) We will suppose that the 
data set « to be chosen at random from all possible sets so that 
one cannot expect to find much redundant structure within it. 
Then the ordered data set requires about b ■ 2* bits of binary 
information for complete description. We won't, however* be in- 
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Pig lire 12.10 



tcrc*tcd in ihc order of ihe words in the data let This reduces 
ihe amount of information required to store the set to about 
(6 - j) - 2* bit*, 

Wc want a machine lhat, when given a random 6-digit word **, 
will answer 

question Mm» In t he data set?* 

and we want to formulaic constraints upon how this machine 
wnrks in such a way that wc can separate computational Aspects 
from memory aspect*. The following scheme achieves this goal 
well enough to show, by examples, how little is known about the 
conjugacy between time and memory. 

Wc will give our machine a memory of Sf separate bit*- -thai is, 
onc-digil binary word*. Wc arc required to compose -in advance, 
before we see the data set two algorithms v . and \<„t that 
satisfy the following conditions: 

I. A* w i* giveo the djia wi U*inj vhi^ as d»ta ( it fills the U bits of 
meBMry tviih informal ion. Nrilhtr ihe dnia set nor A u * arc used again, 
nor u A tnM allowed to gel any information about wtui A n * did. except by 
LRtpVCtinp ihc COnllMlll uf SI. 



' Wc u =l: fxl lt> vni^r imii } in iitHHll fln«rt miltultt 



is*-*. 
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2 A| nd i% then given a random word, v f and asked to answer Question 1. 
using ihc information stored in the memory by A^, We are interested 
in how many bits A,^ hat to consult in the process, 

J. The goal i* lo optimize the design of A* B and Ar„i to minimize 
ihe number of memory references in the question -answering compula- 
tion, averaged over all possible words w* 

I2.4LI Case 1: Enormous Memory 

H is plausible that the larger be M t the smaller will be the average 

number of memory-references Ar*«must make. Suppose that 

M > 2\ 

Let m t be the ith bit in memory; then there is a bit m w for each 
possible query word *\ and we can define 

(A 1L ,. $clm„ to I if *• is in the data set 
A -M : ir is in thcdatascl ifm, m I. 

Thui. with a huge enough memory, only one reference is required 
I o answer Question I. 

1 2.6.2 Ctibe2: Inadequate Memory 
Suppose that 

M < (6 - a)2\ 

Merc, the problem cannot be solved at all, since A fit cannot store 
enough information to describe the data set in sufficient detail* 

!2i6JtCMr3: Binary Logarithmic Sort 
Suppose that 

At - *-2*. 

Now there is enough room to store the ordered data set* Define 

X|*^ siarc the words of the dau set in ascending numerical 

order. 
■V.i*i" perform a binary search to see first which half of memory 

might contain w\ then which qunrtitc, etc. 
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This "ill require at most & ■ log 2* inspections of 6-bit words* 
that is, a * b bit-inspections. 

This U not an optimal March since, (1) otic doc* not always need to 
intpcci a whofe word to decide which word to inspect next, and (2) U 
docs not exploit the uniformity or distribution that the flrxt fl digits of 
the tudcfrJ data *ct will (on (He average) show. Effect I reduces the 
required number from a * n lo ihe order or ) a • b and effect 2 reduces 
ll from o * A <*> o • (6 - tf) T We don't knew exactly how lhc*c two 
effects combine. 

12.6.-4 CMM; Exhaustive Search 
Consider 

M - (6 - *>2'. 

This gives just about enough memory to represent the unordered 
data set. For example we could define 

A*,: First put the words of the dam *ei in numerical order- 
Then compute their successive differences. These will re- 
quire obout (A - *?) bits each Use a standard method of 
information theory. Huffman Coding (say), to represent 
this sequence; it will require about {b - a) 2" bits. 

But the only retrieval schemes Wi can think of ore like 

A ( »g: Add up successive differences in memory until the sum 
equals or exceeds w. If equality occurs, then w is in the 
data set. 

And ihi* requires — *J(6 - o)2* memory references, on (he aver- 
age. It seems eJe+ir that, given this limit on memory, no A** - 
Afttopaircan do much better. That is* we suspect that 

If no exiftt memory is amifahle then, to answer Question /. one 
Mian on the mvrojr<\ uwrf f through half the memory. 

One might go slightly further; even Huffman Coding needs some 
CJUni memory, and if I here is none, A ft can only store an efficient 
"number" for the whole data set. Then the conjecture is that A.,j 
mu.ti jlmo>i always look at almost all of memory. 
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12.6,8 Cas*5: Hash Coding 
Coniider 



U 



2. 



Here we have a case in which there is o substantia) margin of 
extra memory -about twice what is necessary for ihc data set. 
The result here is really remarkable — one might even say counter- 
intuitive — because the mean number of references becomes very 
small* The procedure uses a concept well Vnown to programmers 
who use it in "symbolic assembly programs" for symbol-table 
references, but docs not seem to be widely known to other com* 
putcr "specialists/* It is called hash-coding. 

There arc many variant* or Ihi* idea. We ditcuss a particular form 
adapted to u redundancy of two. 

In the hash-coding procedure. A,.* is equipped with a subprogram 

*{*'•» Uwti S-vcn an integer/ and a fr-bil word w t produces an 
{a + O-bhwofd. The function fl(wj) is "pseudorandom" in the 
scn*< that for cacti y» ff(n\;> maps the set of all 2* input words 
with uniform density on the 2*~ l possible output words and. for 
different y's. these mappings are reasonably independent or 
orthogonal. One could u*e symmetric functions, modular 
arithmetics, or any of the conventional pseudorandom methods** 

Now \ we think ofthefr * 2 *bft memory as organized into fr-bit 
registers with (a + l)-bit addresses: Suppose thai A fl4f has already 
filed the words w » w and it is about to dispose of W**t* 

Am* Compute /?(***■), I). If the register with this address is 
empty put *r, t i in it. If that register is occupied do the 
same with R(m\, j. 2). ft(n* # , T# 3). . < . until an unoccupied 
register A(*i',,i,y)is found: file -v.. ( therein, 

^fipjt Compute ff(*\ I), ff this register contains W» then W is in 
the data set. If R{*\ l> is empty, then n* is not in the date 
*et If A(tr. I) contains some oihcr word not w, then do 



■ Tbtit rt a tupvr *tiiii*it ihiil lt{*. A require* «mt macicut property that can onty 
btf jpcnv* Muted, tl j* Hue lhi»i ony fiirtkCvbr H wilt S* tail on fatihtiktr -Ul* 
Ml *• but lhcf< U no pn^bWm tit all *Wen we citnuder avtfugc t*chivk»f 0« aff 
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the same with R{h\ 2) t and if necessary R(w t 3), R(w, 4), 
. . . » until cither x f or an empty register is discovered. 

On the average. A llt will wake less than 2b memory-hit references} 
To sec this, we note first that, on the average, this procedure leads 
to the inspection pT jti^i 2 registers! For half of the registers are 
empty, and the successive values of R(wJ) for J ■ 1.2, .„ are 
independent (with respect to the ensemble of nil data sets) so the 
mean number of registers inspected to find an empty register is 2. 

Actually, the mejn urmirution lime is slightly /m, because for i/» In 
the d.iia id the c* peeled number of inspected reenters fa < 2. The 
procedure i* useful for symbol*(;ibtcs and the like, wheic one may want 
not only to know if m is there, but also to retrieve some other data as* 
jocmied (pcrhapt again by hash-coding) with it. 

When the margin of redundancy is narrowed, for example, if 



<W - 



4 - I 



*' t\ 



then only (I/Nth) of the celts will be empty and one can expect to 
have to inspect about n registers. 

Because people are accustomed to the fact that most computers 
arc "word-orienicd" and normally do inspect b bits at each 
memory cycle the following analysis has not (to our knowledge) 
been carried through to the case of 1-bit words. When we pro- 
gram A M to match words bit by bit we find that, since half 
the words in memory are zero, matching can be speeded up by 
assigning a special "zero" bit to each word. 

Assume, for the moment, that we have room for these 2* extra 
bits. Now suppose that a certain * j is nut in the data set. (This 
has probability W - "*) Fir&l inspect the "zero" hit associated 
wiih /T(W(hJ). This Ins proh; iti, lity J of being zero, [fit is not zero, 
then we match the bits of w$ with those of the word associated 
with ff(H>o,l). These cannot all match (since **> isn't in the data 
set) and in fact the mismatch will be found in (an average of) 
2 ■ I + J + J + * . - references. Then the "zero" bit of /?(h'o,2) 
must be inspected, and the process repeats. Each repeat has prob- 
ability J and ihe process terminates when the "zero" bit of some 
Rt w +J) ■ 0* The expected number of references con be counted 
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then at 

1(1+2 + 4(1+2+ )(♦..») + I - 3 + I - 4. 
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If u*0 win the Jala set (an event whose probability is 2*"*) and we 
repeal the analysis we gel 4 + b reference*, because the process 
musttenninateby matching all 6 bits of H>p, 

The expected number of references, overall^ is then 



4(1 - 2"*) + (4 + 6)2-*- 4 

- 4 



+ 6*2 



i-t 
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since normally 2*'* will be quite liny* Wc consider it quite re* 
markablc that so little redundancy— a factor of two— yields this 
small number! 

The estimate* above arc on the hijh side because in the case that *t is 
hi the data set the "run length" through fl(>*w) will be shorter, by 

( nearly a factor of J, than chance would have it just because they were put 
there by A ftfc . On the other hand* we must pay for the extra "*ero" 
bits we adjoined to A/. If wc have M - 26*2 bits and make worts 
of length b + I instead of 6 P then the memory* becomes slightly more 
than half full: in fact, we must replace "4" in our rctuti by something 
Nic4|{6 + l)/(fr - I)). Perhaps these two effects will offset one another; 
we haven't matte exact calculations, mainly because we arc not sure that 
even this Apj* Am pair is optimal* 

It certain!) seems suspicious that half the words in memory are simply 
empty! On Ihc other band, the best one could expect from further im- 
proving the algorithms it to replace 4 by 3 (or 2?), and this Is not a targe 
enough carrot to work hard for. 

12*6.6 Summary of Exact Marching Algorithm* 
To summarize our results on Question I we have established 
upper bounds for the following cases: Wc believe that they arc 
close I o lower bounds also but, especially in cases 3 and 4, arc 
not sure. 



Case Memory si« Bit- references 



Method 



* 



2 


<(b - a)2' 


« 


(impossible) 


4 


{b - a)2' 


J(6-fl)2* 


(search all memory) 


3 


b-2* 


}b'S 


(logarithmic son) 


5 


2b -2' 


4 + € 


(hash coding) 


1 


il> 


1 


(table look-up) 
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12.7 Time *s. Memory for Best Matching: An Open Problem 
We have summarized our (limited) understanding of "Question I" 
— the exact matching problem— by ihc little table in §12.6*6. IT 
one "plots the curve' 4 one is instantly struck by the effectiveness 
of small degrees of redundancy. We do not believe that this 
should be taken loo seriously, for we suspect that when the prob- 
lem is slightly changed the result may be quite different* We con* 
sidcr now 

UUEsriON 2: Given ^. exhibit the word A closest to w in the data 

The ground roles about A*, and Am arc the same, and distance 
can be chosen to be the usual metric* that U, (he number of digits 

in which two words disagree. If Xi,.. *i*» and £ , S* arc the 

(binary) coordinates of points w and w then we define the 
Hamming distance to be 



3t 

rf(»-.*)-Ek- u 
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One gets exactly the same situation with the usual Cartesian 
distance C{h\ »*■), because 

so both C(«\ 9) and <ft>\ *>) arc minimized by the same *. 

12.7.1 Case UMm 2*-*- 

Af* assigns Tor every possible word w a block of* bits that con- 
tain the appropriate bits of the correct w, 

A ( „i looks in the block for w and writes out f&, It uses 6 references, 
which is obviously the smallest possible number. 

12.7.2 Case 2: A/ < (b - a)2\ 
Impossible* for same reason as in Question 1. 

l2JJCaseJ:M - ft-2* 
No result known. 

lt$AGm4*M - (b * a)2* 

This presumably requires {b - <j)-2* references* that is t all of 

memory, for the tame reason as in Question \. 
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12.7.5 Ca**5:(fc - a)2- < M <b-2* 
No useful results known. 

12-7.6 Glooaiy Prospects for Best Matching Algorithms 
The results in $12.6.6 showed that relatively small factors of re- 
dundancy in memory siic yield very large increases in speed, for 
serial computations requiring the discovery of exact matches. 
Thus, (here is no great advantage in using parallel computational 
mechanisms. In fact, as shown in §126,5, a mcmory-*>« factor of 
just 2 is enough to reduce the mean retrieval lime to only slightly 
more than the best possible. 

But. when we turn to (he hesi Match problem, alt this seems to 
evaporate. In fact, we conjecture that even for the best possible 
Ato-A^ pairs, th e speed-up value of large mem o ry re dundancies 
is very smal Taml for large data sets with long w o rd lengths there 
are no practical alternatives to large searches thai inspect large 
pa r u of the memory . 

We apologize for not having a more precise statement of the con* 
jecture. or good suggestions for how to prove it, for we feel that 
this is a fundamentally important point in the theory of computa- 
tion* especially in clarifying the distinction between serial and 
pir.iHel concepts* 

Our belief in this conjecture is based in part on experience in find- 
ing fallacies in schemes proposed for constructing fast bcsi- 
maiching file and retrieval algorithms. To illustrate this wc discuss 
nc\t the proposal most commonly advanced by students. 

12.7.7 The Numerical-Order Scheme 

This proposal is a natural attempt to extend the method of Case 3 

(12.6,3) from exact match to best match. The scheme is 

Ami " storetlicwordsorthedatasetin numerical order 
A Af ,j'. given a word W, find (by some procedure) those words whose 
first (T bits agree most closely with the first a bits of w. How 
to 4o this isn't clear, but it occurs to one that (since this is 
the same problem on a smaller scale?) the procedure could 
be recursively defined. Then sec how well ihe other bits of 
these words mulch wiih h-. Next, ...(?)-, .. 

The intuiiive idea is simple: the word in the data set that is 
clownt to ** ought to show bcitcMhan*chancc agreement in the 
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first a bits, so why not look fust at words known to have this 
properly* There arc two disastrous bugs in this program: 

U When can one stop searching? What should we fill in where we 

wrote "Next -..(?) " We know no nontrivial rule thai 

guaranitei getting the best match. 

2, The intuitive concept, reasonable as it may appear, is not 
valid? It isn't even of much use for rinding a good match, let alone 
rinding the best match. 

To elaborate on point 2* consider an example: Tci a - 20, b ■ 10.000. 
Let M, for simplicity, be the all-jcro word. A typical word ui the data 
*ei uill ha*e a mean of 5000 flue's, and 5000 wo's. The standard 
deviauonwillbeJ(l0,0u0) p " - 50, Thus, kss than one word in 2' - 2* 
can be expected to have fewer than 4750 one*&. Hence, the closest word 
in the 'l - 1 set will (on the average) have at leaii this many otw*%. That 
closest word will hove (on the average) >20*(47iO/l0,000) - 9.5 Oftfs 
among its first 20 bits! The probability that w will indeed have very 
lew owe's in its first 20 bits is therefore extremely tow, and the slight 
Favorable bias obtained by inspecting thest words firri is quite utterly 
nefllij ible in reducing the amount of inspection. Besides, objection 1 siill 
Kami 

The value of ordering the first few bits of the word* is quite usdeU. 
then, Clu&xirving word*, in this way amounts, in the ^dimensional 
geometry, to breaking up the space into "cylinders" which ore not well 
shaped for finding nearby points. We have, therefore, tried various ar- 
r jng emenu of spheres, but the same sons of trouble appear (after more 
anatyibp. In the course of that analysis, we are led to suspect that there 
bi fundamental properly of i<dimcniiona1 geometry that puts a very 
iironjt and discouraging limitation upon all such algorithm*. 
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12,7,8 Why U Besi March so Different from FNaci Match? 
If our unproved conjecture is true, one might want at least an 
intuitive explanation for the difference we would get between 
512.6.3 and 12,7,3. One wa> to look at it is to emphasize thit, 
though the phrase* * l l>ett match** and "exact match" sound simi- 
lar to the ear. they really are very different. For in the case of 
caiici match, no error is allowed, and this has the remarkable 
effect of changing an n*dinixmtenal prohtent into a one+dimensionol 
pmhtem! For best matching we used the formula 

Error - £ k - i,| - £ 1 |x, - ftl, 
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where we have inserted Lhe coefficient *M" to show that all errors, 
in different dimension!, arc counted equally* Bui for exact match, 
since no error can be tolerated, we don 1 ! have to weight them 
equally: any positive weights will do? So for exact match we could 
just as well write 

Error - £ *'!*< " -M *' evc " Error - J^ 2'(x t - x t ) 



j- i 



f-i 



because cither of these can be zero only when all x t * v L (Shades 
of stratification.) But then weean (finally) rewrite the latter as 

Error - (S2'x,) - (X2%) 

and we have mapped the ^-dimensional vector (x x*) into a 

single point on a one -dimensional line. Thus these superficially 
similar problems have totally different mathematical personali- 
ties! 

12. fl Incremental Compulation 

All the Ate algorithms mentioned have the following curiously 
local property. They can be described roughly as computing the 
slorcd information M as a function of a large data set: 

M - A,, (data set) , 



Now one can imagine algorithms which would use a vast amount 
of temporary storage (that is, much more than W or much more 
than Is needed to store the data set) in order to compute M, Our 
A «* algorithms do not. On the contrary, they do not even use 
significantly more memory capacity than is needed to hold their 
final output, Jf. They are even conlcnUo examine just one mem- 
ber of the data set at a time, with no control over which they 
will sec ncxL and wilhoul any subterfuge of storing the data in* 
icrnulty. 

It seems to ttftlhat this is an interesting properly of compulation 
that deserves to be studied In its own right. It is striking how 
many apparently "global" properties of a data set can be com- 
puted "incrementally" In this sense. Ralhcr than give formal 
definitions of these terms, we shall illustrate them by simple 
examples. 
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ihc dtfi is picstMcd only once, 

Z pass" SmSfC '*• lhe MCond smallCSl °" fc " * ? 5 » 

the meJiiin* 

|, might asm II M .*ta. »»*«*. lhat an STT!!!lCI 

am 

awcpublc probability orerror. 

seiis: 

|lhe number in ihc data «, «nctc««cd in numerically 

increasing **«■» form a P"'™ number]. 

decision, about wheihcr concalcnalmns of members ol 
sot ate balling tapes for Turing machines. 
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